Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 2105003: on-prem: improvements on resolv-prepender #3271

Merged
merged 5 commits into from
Aug 11, 2022

Conversation

jcaamano
Copy link
Contributor

@jcaamano jcaamano commented Jul 28, 2022

Work around several resolv-prepender issues:

  • hostnamectl fails due to being denied by selinux
  • resolv-prepender waits more than the overall NM timeout, making devices to not come up
  • resolv-prepender copies resolv.conf without nameservers
  • kubelet starts before resolve-prepender has had a chance to set resolv.conf and openshift-dns coredns pods fail to start.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 28, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 28, 2022

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot requested review from creydr and mdbooth July 28, 2022 10:50
@jcaamano jcaamano force-pushed the wait-resolv-conf branch 6 times, most recently from da83d68 to 57daeab Compare August 1, 2022 18:42
@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 1, 2022

/test ?

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 1, 2022

@jcaamano: The following commands are available to trigger required jobs:

  • /test 4.11-upgrade-from-stable-4.10-images
  • /test cluster-bootimages
  • /test e2e-agnostic-upgrade
  • /test e2e-aws
  • /test e2e-gcp-op
  • /test images
  • /test unit
  • /test verify

The following commands are available to trigger optional jobs:

  • /test 4.11-upgrade-from-stable-4.10-e2e-aws-ovn-upgrade
  • /test bootstrap-unit
  • /test e2e-aws-disruptive
  • /test e2e-aws-ovn-workers-rhel8
  • /test e2e-aws-proxy
  • /test e2e-aws-serial
  • /test e2e-aws-single-node
  • /test e2e-aws-upgrade
  • /test e2e-aws-upgrade-single-node
  • /test e2e-aws-workers-rhel8
  • /test e2e-azure
  • /test e2e-azure-ovn-upgrade
  • /test e2e-azure-upgrade
  • /test e2e-gcp-op-single-node
  • /test e2e-gcp-single-node
  • /test e2e-gcp-upgrade
  • /test e2e-metal-assisted
  • /test e2e-metal-ipi
  • /test e2e-metal-ipi-ovn-dualstack
  • /test e2e-metal-ipi-ovn-ipv6
  • /test e2e-openstack
  • /test e2e-openstack-parallel
  • /test e2e-ovirt
  • /test e2e-ovirt-upgrade
  • /test e2e-ovn-step-registry
  • /test e2e-vsphere
  • /test e2e-vsphere-upgrade
  • /test e2e-vsphere-upi
  • /test okd-e2e-aws
  • /test okd-e2e-gcp-op
  • /test okd-e2e-upgrade
  • /test okd-e2e-vsphere
  • /test okd-images

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-machine-config-operator-master-bootstrap-unit
  • pull-ci-openshift-machine-config-operator-master-e2e-agnostic-upgrade
  • pull-ci-openshift-machine-config-operator-master-e2e-aws
  • pull-ci-openshift-machine-config-operator-master-e2e-gcp-op
  • pull-ci-openshift-machine-config-operator-master-e2e-openstack
  • pull-ci-openshift-machine-config-operator-master-images
  • pull-ci-openshift-machine-config-operator-master-okd-images
  • pull-ci-openshift-machine-config-operator-master-unit
  • pull-ci-openshift-machine-config-operator-master-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 1, 2022

/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6

1 similar comment
@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 1, 2022

/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6

@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 2, 2022

/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6

@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 2, 2022

/test e2e-metal-ipi-ovn-dualstack

1 similar comment
@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 8, 2022

/test e2e-metal-ipi-ovn-dualstack

@jcaamano jcaamano force-pushed the wait-resolv-conf branch 2 times, most recently from 066a2fb to 3fa18a3 Compare August 9, 2022 09:27
@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 9, 2022

/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-ipv6
/test e2e-metal-ipi

@jcaamano
Copy link
Contributor Author

jcaamano commented Aug 9, 2022

/test e2e-metal-ipi-ovn-dualstack

1 similar comment
@jcaamano
Copy link
Contributor Author

/test e2e-metal-ipi-ovn-dualstack

@jcaamano
Copy link
Contributor Author

/test e2e-metal-ipi-ovn-ipv6
/test e2e-metal-ipi

Currently a NetworkManager dispatcher script does not have the correct
selinux permission to dbus chat with hostnamed. Work around the issue
using systemd-run.

See: https://bugzilla.redhat.com/show_bug.cgi?id=2111632

Signed-off-by: Jaime Caamaño Ruiz <[email protected]>
If resolve prepender takes more than NetworkManager timeout, currently
90s, it might fail to bring up devices before we had a chance to
process all possible events for a device. This needs to account for
different type of events and IPv4 and IPv6 events in case of dual stack
and overall take less then the NetworkManager timeout.

Signed-off-by: Jaime Caamaño Ruiz <[email protected]>
node-ip can fail if a device is not ready to be bound to. Retry but
don't add to the overall timeout more than the NetworkManager timeout
(90s) accounting for all the events we need to attend to.

Signed-off-by: Jaime Caamaño Ruiz <[email protected]>
Make resolv-prepender wait for nameservers in
/run/NetworkManager/resolv.conf in all cases to avoid copying it without
them to /etc/resolv.conf

Signed-off-by: Jaime Caamaño Ruiz <[email protected]>
Without a properly configured resolv.conf, openshift-dns coredns will
fail to run. These pods have a default DNS policy and will use the host
resolv.conf, which is the one kubelet gets when it starts.

Signed-off-by: Jaime Caamaño Ruiz <[email protected]>
@openshift-ci openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Aug 10, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 10, 2022

@jcaamano: This pull request references Bugzilla bug 2105003, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.12.0) matches configured target release for branch (4.12.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

In response to this:

Bug 2105003: on-prem: improvements on resolv-prepender

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jcaamano
Copy link
Contributor Author

/test e2e-metal-ipi
/test e2e-vsphere
/test e2e-ovirt

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 10, 2022

@jcaamano: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@cybertron
Copy link
Member

/lgtm

This looks awesome, thanks!

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 10, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 11, 2022

@jcaamano: This pull request references Bugzilla bug 2105003, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.12.0) matches configured target release for branch (4.12.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @sunzhaohua2

In response to this:

Bug 2105003: on-prem: improvements on resolv-prepender

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jcaamano
Copy link
Contributor Author

/assign @sinnykumari

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 11, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cybertron, jcaamano, sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 11, 2022
@openshift-merge-robot openshift-merge-robot merged commit dcb00e0 into openshift:master Aug 11, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 11, 2022

@jcaamano: All pull requests linked via external trackers have merged:

Bugzilla bug 2105003 has been moved to the MODIFIED state.

In response to this:

Bug 2105003: on-prem: improvements on resolv-prepender

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@mandre
Copy link
Member

mandre commented Aug 16, 2022

/cherry-pick release-4.11

@openshift-cherrypick-robot

@mandre: new pull request created: #3287

In response to this:

/cherry-pick release-4.11

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mandre added a commit to shiftstack/machine-config-operator that referenced this pull request Aug 31, 2022
In order to avoid a podman issue [1] causing a layer corruption when an
image pull is killed midway, let's move the image pull outside of the
timeout command.

The timeout was recently reduced to 20 seconds with [2] making the issue
more likely to happen.

[1] containers/podman#14003
[2] openshift#3271
mandre added a commit to shiftstack/machine-config-operator that referenced this pull request Aug 31, 2022
In order to avoid a podman issue [1] causing a layer corruption when an
image pull is killed midway, let's move the image pull outside of the
timeout command.

The timeout was recently reduced to 20 seconds with [2] making the issue
more likely to happen.

[1] containers/podman#14003
[2] openshift#3271
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants